Skip to content

feat(plan-feature): add testing task generation from readiness template#226

Merged
mrizzi merged 1 commit into
mainfrom
TC-4878
Jul 2, 2026
Merged

feat(plan-feature): add testing task generation from readiness template#226
mrizzi merged 1 commit into
mainfrom
TC-4878

Conversation

@mrizzi

@mrizzi mrizzi commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add testing readiness template discovery (docs/testing-readiness.md) to plan-feature Step 3
  • Add testing task generation to plan-feature Step 5 — one task per test category in the template
  • Add 3 new constraints (§1.85–1.87) to docs/constraints.md with traceability index update
  • Create eval fixture with 2 test categories (Smoke Tests, Performance Benchmarks) and add positive/negative eval assertions

Implements TC-4878

Test plan

  • Verify eval 1 assertions pass with testing-readiness.md fixture (positive case: 2 testing tasks generated)
  • Verify eval 2 assertions pass without testing-readiness.md fixture (negative case: no testing tasks)
  • Verify constraints §1.85–1.87 trace to SKILL.md Step 5 Testing task generation
  • Verify Traceability Index includes the new constraints

🤖 Generated with Claude Code

Summary by Sourcery

Add support in the plan-feature skill for generating testing tasks based on a repository testing readiness template and document the corresponding constraints and traceability.

New Features:

  • Introduce testing readiness template discovery in plan-feature to read test categories from docs/testing-readiness.md per target repository.
  • Add testing task generation in plan-feature Step 5, creating one testing task per test category defined in the readiness template.

Documentation:

  • Document new constraints governing testing task generation and dependencies in constraints.md, with updated traceability index to reference the new plan-feature instructions.

Tests:

  • Add an eval testing-readiness.md fixture with example test categories to support positive and negative evaluation of testing task generation behavior.

Add auto-discovery of docs/testing-readiness.md in target repositories
during Step 3, and generate one cross-cutting testing task per test
category in the template during Step 5. Testing tasks depend on all
implementation tasks and omit code-change sections.

Includes 3 new constraints (§1.85–1.87) with traceability, eval fixture
with 2 test categories, and positive/negative eval assertions.

Implements TC-4878

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Assisted-by: Claude Code
@sourcery-ai

sourcery-ai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Adds testing readiness template discovery to plan-feature Step 3, introduces testing task generation to Step 5 backed by new constraints, and wires in an eval fixture and traceability updates to validate positive/negative behavior.

Flow diagram for testing readiness discovery and testing task generation

flowchart LR
  subgraph Step3_Testing_readiness_template_discovery
    A[Get repository_root from Repository_Registry]
    B{docs/testing-readiness.md exists?}
    A --> B
    B -->|yes| C[Read testing-readiness.md]
    C --> D[Parse headings as test_categories]
    D --> E[Store test_categories for Step5]
    B -->|no| F[Log silently and continue]
  end

  subgraph Step5_Task_generation
    G[Generate implementation_tasks]
    H{test_categories available?}
    G --> H
    H -->|no| I[Skip testing_task generation]
    H -->|yes| J[For each test_category generate testing_task]
    J --> K[Set testing_task.dependencies = all implementation_tasks]
  end

  Step3_Testing_readiness_template_discovery --> Step5_Task_generation
Loading

File-Level Changes

Change Details Files
Introduce testing readiness template discovery as part of plan-feature Step 3 and persistence of parsed categories for later task generation.
  • Add a new Step 3 subsection describing how to resolve repository root from CLAUDE.md registry.
  • Specify lookup of docs/testing-readiness.md using Glob/Read list_dir mechanisms and silent handling when absent.
  • Define parsing rules for ## headings and bullet acceptance criteria, and storing parsed categories for Step 5 use.
plugins/sdlc-workflow/skills/plan-feature/SKILL.md
Define testing task generation behavior in Step 5 using the readiness template and task-description-template.md conventions.
  • Add Step 5 instructions to gate testing task generation on presence of a readiness template discovered in Step 3.
  • Specify that one testing task is generated per test category heading with fields (Repository, Target Branch, Description, Acceptance Criteria, Test Requirements, Dependencies) derived from the template and feature issue.
  • Require testing tasks to depend on all implementation tasks and explicitly omit modification-centric sections (Files to Modify/Create, API Changes, Implementation Notes, Reuse Candidates, Verification Commands).
plugins/sdlc-workflow/skills/plan-feature/SKILL.md
Add formal constraints and traceability entries for testing task generation behavior.
  • Introduce constraints §1.85–1.87 capturing mandatory testing task generation, non-generation when no template, and dependency on implementation tasks.
  • Update the SKILL.md source mapping line to include Step 5 Testing task generation with references to §1.85–1.87.
docs/constraints.md
Add an eval fixture testing-readiness template to exercise positive/negative testing task generation behavior.
  • Create a testing-readiness.md fixture that defines two test categories (Smoke Tests, Performance Benchmarks) with bullet acceptance criteria.
  • Wire the fixture into plan-feature evals (positive case with two testing tasks, negative case without template) and add assertions around task generation behavior.
evals/plan-feature/evals.json
evals/plan-feature/files/testing-readiness.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The testing readiness template discovery and task generation logic could better specify behavior when some target repositories have docs/testing-readiness.md and others do not (e.g., per-repository task generation vs. all-or-nothing), to avoid ambiguity with constraints §1.85–1.86.
  • Consider defining how to handle edge cases in docs/testing-readiness.md parsing, such as missing ## headings, empty categories, or duplicate category names, so the skill’s behavior remains predictable with imperfect templates.
  • The requirement that testing tasks omit Verification Commands may make it harder to indicate how tests should actually be run; you might want to clarify where execution details should live for these tasks to keep the workflow consistent.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The testing readiness template discovery and task generation logic could better specify behavior when some target repositories have `docs/testing-readiness.md` and others do not (e.g., per-repository task generation vs. all-or-nothing), to avoid ambiguity with constraints §1.85–1.86.
- Consider defining how to handle edge cases in `docs/testing-readiness.md` parsing, such as missing `##` headings, empty categories, or duplicate category names, so the skill’s behavior remains predictable with imperfect templates.
- The requirement that testing tasks omit Verification Commands may make it harder to indicate how tests should actually be run; you might want to clarify where execution details should live for these tasks to keep the workflow consistent.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eval Results

Eval Results: plan-feature

Eval Passed Failed Pass Rate
eval-1 14/19 5 74%
eval-2 12/16 4 75%
eval-3 11/15 4 73%
eval-4 6/11 5 55%
eval-5 10/15 5 67%
eval-6 5/14 9 36%

Failed Assertions

eval-1: 5 failing assertions
  • Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
    Evidence: "Searched all 9 output files for 'sha256', 'digest', and 'sha-256' — no matches found. No evidence of description digest comments in any output file. The outputs contain only the task description files and impact map with no digest markers or comment records."

  • Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
    Evidence: "Searched all output files for 'convention', 'CONVENTIONS', and 'Applies:' — found only incidental uses of the word 'conventions' in task-1 ('field conventions') and task-3 ('repo conventions'), neither of which are convention applicability rationales in the prescribed format. No task contains 'Per CONVENTIONS.md' references or 'Applies: task modifies <file> matching the convention's <scope>' rationales as required by convention-applicability-rules.md. The repo-backend.md manifest lists a CONVENTIONS.md file, so conventions should have been evaluated for applicability."

  • Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
    Evidence: "Feature fixture (feature-standard.md) has 'Priority: Major'. Searched all output files for 'priority', 'Major', and 'additional_fields' — no matches found. No output file contains evidence that priority was propagated to tasks via additional_fields."

  • Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
    Evidence: "Feature fixture has 'Fix Versions: RHTPA 1.5.0'. CLAUDE.md has no '### Jira Field Defaults' section, so fixVersion scope defaults to 'both', meaning fixVersions should be propagated to tasks. Searched all output files for 'fixVersion', 'RHTPA', and 'additional_fields' — no matches found. No output file contains evidence that fixVersions was propagated to tasks."

  • Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
    Evidence: "No summary comment file exists in the outputs directory. The outputs contain only: impact-map.md and task-1 through task-8 description files. Searched for 'summary comment', 'propagat', 'step 6c' — no matches found. No evidence of a summary comment being generated or posted."

eval-2: 4 failing assertions
  • Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
    Evidence: "No sha256 digest, hash, or '[sdlc-workflow] Description digest' marker appears anywhere in the 5 output files (impact-map.md, task-1 through task-4). grep for 'sha256', 'digest', and 'hash' across all output files returned no matches. No separate digest or comment files exist in the outputs directory."

  • Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
    Evidence: "The feature has Priority: Normal. No output file contains any reference to 'priority', 'additional_fields', or 'Normal'. grep for 'priority' across all output files returned no matches. There is no evidence that priority was included in any task creation parameters."

  • Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
    Evidence: "The feature has Fix Versions: RHTPA 1.6.0, and CLAUDE.md has no Jira Field Defaults section (defaulting to 'both'), so fixVersions should be propagated. No output file contains any reference to 'fixVersion', 'RHTPA 1.6.0', or 'additional_fields'. grep for 'fixversion' (case-insensitive) across all output files returned no matches."

  • Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
    Evidence: "No summary comment file exists in the outputs directory. The only files are impact-map.md and task-1 through task-4.md. None of these files contain a summary comment referencing priority inheritance, fixVersion propagation, or any Step 6c content."

eval-3: 4 failing assertions
  • Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
    Evidence: "No output file contains any SHA-256 digest, '[sdlc-workflow] Description digest:' marker, 'sha256-md:', or 'sha256-adf:' string. The outputs directory contains only 9 files (impact-map.md, task-1 through task-8 markdown files) — none of which include evidence of digest comments being posted after task creation."

  • Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
    Evidence: "The Feature issue has Priority: Critical, so every task should inherit priority: Critical in additional_fields. No output file contains any mention of 'additional_fields', 'priority: Critical', or any evidence that the priority field was propagated to the created tasks. The task description files contain only markdown descriptions without Jira API field metadata."

  • Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
    Evidence: "The Feature issue has Fix Versions: RHTPA 1.5.0, and CLAUDE.md has no Jira Field Defaults section (defaulting scope to 'both'), so every task should include fixVersions: [RHTPA 1.5.0] in additional_fields. No output file contains any mention of 'fixVersions', 'RHTPA 1.5.0', 'additional_fields', or any evidence that the fix version was propagated to the created tasks."

  • Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
    Evidence: "No output file contains a summary comment for the feature issue. There is no file referencing Step 6c, no summary of propagated priority/fixVersion values, and no explanation for their omission. The outputs directory contains only the impact map and 8 task description files — no summary comment file exists."

eval-4: 5 failing assertions
  • Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
    Evidence: "grep for 'sha256', 'digest', and 'Description digest' across all output files returned NO MATCHES. None of the 5 output files (impact-map.md, task-1 through task-4) contain any SHA-256 hash, digest comment, or '[sdlc-workflow] Description digest:' marker. No separate comment files exist in the outputs directory."

  • Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
    Evidence: "No convention-enrichment entries using the prescribed format appear in any task description. Tasks 1-3 reference repository patterns with free-form prose (task-1 line 30: 'Per the Key Conventions in the repository: all error handling should use...'; task-2 line 61: 'Per repository conventions: all handlers return...'; task-3 line 36: 'Per repository conventions: integration tests hit...') but none use the prescribed 'Applies: task modifies <file> matching the convention's <scope>' rationale format. No 'Per CONVENTIONS.md §...' references appear anywhere. While no inapplicable conventions are annotated with 'Not applicable' (satisfying that part), the convention references that do exist use free-form prose instead of the prescribed format, and no evidence of formal convention-applicability validation exists."

  • Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
    Evidence: "The feature fixture (feature-adversarial.md line 6) sets Priority: Major. grep for 'priority' (case-insensitive) across all task-*.md files returned NO MATCHES. None of the 4 task descriptions contain an 'additional_fields' section or any reference to priority inheritance. The priority 'Major' from the feature was not propagated to any task."

  • Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
    Evidence: "The feature fixture (feature-adversarial.md line 7) sets Fix Versions: RHTPA 1.5.0. The CLAUDE.md project config has no '### Jira Field Defaults' section, so fixVersion scope defaults to 'both', meaning tasks should inherit fixVersions. grep for 'fixversion' and 'fix.version' (case-insensitive) across all task-*.md files returned NO MATCHES. None of the 4 task descriptions contain an 'additional_fields' section or any reference to fixVersion inheritance. The fixVersion 'RHTPA 1.5.0' from the feature was not propagated to any task."

  • Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
    Evidence: "No summary comment file exists in the outputs directory. The only files present are impact-map.md and task-1 through task-4. grep for 'priority' and 'fixversion' (case-insensitive) returned no matches in any output file. There is no evidence of a summary comment being generated for the feature issue, let alone one that documents priority (Major) and fixVersion (RHTPA 1.5.0) propagation."

eval-5: 5 failing assertions
  • Assertion: "Each non-documentation task file contains all required template sections: Repository, Target Branch, Description, at least one of Files to Modify or Files to Create, Implementation Notes, Acceptance Criteria, Test Requirements. Documentation tasks are exempt from requiring Files to Modify, Files to Create, and Implementation Notes — they must still include Repository, Target Branch, Description, Acceptance Criteria, and Test Requirements"
    Evidence: "Intermediate non-documentation tasks (2-6) all contain the required sections: Repository, Target Branch, Description, Files to Modify/Create, Implementation Notes, Acceptance Criteria, Test Requirements. Documentation task (Task 7) correctly includes Repository, Target Branch, Description, Acceptance Criteria, Test Requirements (exempt from Files to Modify/Create and Implementation Notes). However, the bookend tasks (Task 1 and Task 8) are non-documentation tasks that lack 'Files to Modify'/'Files to Create' and 'Implementation Notes' sections. Task 1 has: Repository, Target Branch, Bookend Type, Description, Acceptance Criteria, Test Requirements, Dependencies — missing Files to Modify/Create and Implementation Notes. Task 8 similarly lacks these sections. The assertion only exempts documentation tasks, not bookend tasks."

  • Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
    Evidence: "No evidence of description digest comments in any output file. The outputs consist of 8 task files and 1 impact-map.md. None contain '[sdlc-workflow] Description digest: sha256-' markers or any SHA-256 hashes. No log files, comment files, or other artifacts showing digest comments were posted after task creation."

  • Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
    Evidence: "The feature issue has 'Priority: High' (feature-feature-branch.md line 8). However, none of the 8 task files contain an 'additional_fields' section or any mention of 'priority: High' being inherited. No task file includes the word 'additional_fields'. The priority value 'High' is not propagated to any task."

  • Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
    Evidence: "The feature issue has 'Fix Versions: RHTPA 2.0.0' (feature-feature-branch.md line 9). CLAUDE.md has no '### Jira Field Defaults' section, so fixVersion scope defaults to 'both', meaning fixVersions should be propagated to all tasks. However, none of the 8 task files contain an 'additional_fields' section or any mention of 'fixVersions' or 'RHTPA 2.0.0'. The fixVersion value is not propagated to any task."

  • Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
    Evidence: "No summary comment file exists in the outputs directory. The outputs contain only 8 task files and 1 impact-map.md. The impact-map.md does not include any mention of priority ('High') or fixVersion ('RHTPA 2.0.0') propagation. No file in the outputs directory contains a summary comment addressing priority or fixVersion inheritance."

eval-6: 8 failing assertions
  • Assertion: "Each task specifies its parent Epic key in the output — tasks targeting trustify-backend are assigned to the backend Epic, tasks targeting trustify-ui are assigned to the frontend Epic"
    Evidence: "No task file (task-1 through task-8) contains any reference to a parent Epic key. The tasks specify a 'Repository' section and 'Dependencies' section, but none mention an Epic key (e.g., TC-XXXX). The impact-map.md assigns tasks to Epics conceptually ('Tasks 1, 2, 3' to Epic 1, 'Tasks 4, 5, 6, 7' to Epic 2), but the individual task files themselves do not specify their parent Epic key."

  • Assertion: "Incorporates links are created from the Feature to each Epic (not from Feature to individual Tasks)"
    Evidence: "No output file contains any reference to 'Incorporates' links, link types, or link creation from the Feature (TC-9006) to any Epic. Searched all output files for 'Incorporates', 'incorporates', and 'link type' with no results."

  • Assertion: "Epics are created with the level-1 issue type name ('Epic') and parent set to the feature issue key"
    Evidence: "No output file contains evidence that Epics were created with issue type 'Epic' or with parent set to TC-9006. The impact-map.md lists 'Epic 1' and 'Epic 2' in a table but does not specify the Jira issue type name or parent field configuration. No Jira API creation details or Epic issue keys are present in any output file."

  • Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
    Evidence: "No output file contains any reference to 'sha256', 'digest', 'Description digest', or '[sdlc-workflow]'. Searched all files in /tmp/plan-feature-eval-pr/eval-6/outputs/ with no matches. No digest comments were posted for any of the 8 tasks."

  • Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
    Evidence: "No task file contains convention applicability rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'). Searched all output files for 'Applies:' and 'convention' — only references found are mentions of 'Conventional Commits format' in Implementation Notes and 'error handling convention', which are free-form prose references to coding patterns, not convention-applicability-rules.md formatted rationales."

  • Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
    Evidence: "The Feature TC-9006 has priority 'Major' (from feature-epic-hierarchy.md). However, no task file contains any reference to 'priority', 'Major', or 'additional_fields'. Searched all output files for these terms with no relevant matches. The priority was not inherited or propagated to any task."

  • Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
    Evidence: "The Feature TC-9006 has fixVersions 'RHTPA 1.5.0' (from feature-epic-hierarchy.md). The CLAUDE.md has no '### Jira Field Defaults' section, so fixVersion scope defaults to 'both', meaning tasks should inherit fixVersions. However, no task file contains any reference to 'fixVersions', 'RHTPA 1.5.0', or 'additional_fields'. The fixVersions were not inherited or propagated to any task."

  • Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
    Evidence: "No output file contains a summary comment for the feature issue. Searched all output files for 'summary comment', 'Step 6', 'priority', 'fixVersion', and 'RHTPA' with no relevant matches. No summary comment was generated or posted on the feature issue."

Pass rate: 63% · Tokens: 68,231 · Duration: 277s

Baseline (be2d4411): 87% · 82,759 tokens · 389s


Generated by sdlc-workflow/run-evals v0.12.0

@mrizzi

mrizzi commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

[sdlc-workflow/verify-pr] Re: @sourcery-ai[bot] review — Classified as suggestion — these propose specification improvements (edge case handling, multi-repo behavior, execution detail placement) that are not backed by documented conventions or established codebase patterns. No sub-task created.

@mrizzi

mrizzi commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

Verification Report for TC-4878 (commit cc97b16)

Check Result Details
Review Feedback PASS 3 suggestions from sourcery-ai[bot]; no code change requests
Root-Cause Investigation N/A No sub-tasks created
Scope Containment PASS 4/4 files match task specification exactly
Diff Size PASS +82/-8 lines across 4 files — proportionate to task scope
Commit Traceability PASS Commit cc97b16 references TC-4878
Sensitive Patterns PASS No secrets or credentials detected
CI Status PASS All 4 checks pass
Acceptance Criteria PASS 8 of 8 criteria met
Test Quality WARN Eval Quality: WARN — see analysis below. Repetitive Test Detection: N/A. Test Documentation: N/A
Test Change Classification N/A No test files in PR
Verification Commands N/A None specified

Overall: PASS

All checks pass. Test Quality WARN is informational and does not affect the overall result.

Eval Quality Analysis

All 5 new assertions introduced by this PR pass (2 testing-task positive assertions in eval-1, 1 testing-task dependency assertion in eval-1, 1 negative assertion in eval-2, plus the TC-4877 documentation assertion in eval-2).

Overall eval pass rate dropped from 87% (baseline be2d4411) to 64% (current). Per-eval breakdown:

Eval Baseline Current Changed by PR?
eval-1 16/16 (100%) 14/19 (74%) Yes — prompt updated, 3 assertions added
eval-2 12/14 (86%) 12/16 (75%) Yes — 1 assertion added (+ 1 from TC-4877)
eval-3 15/15 (100%) 11/15 (73%) No
eval-4 7/11 (64%) 6/11 (55%) No
eval-5 11/15 (73%) 10/15 (67%) No
eval-6 14/14 (100%) 5/14 (36%) No

Regression vs non-determinism: Evals 3–6 were not modified by this PR yet show significant variance — eval-6 dropped from 100% to 36% and eval-3 from 100% to 73% with zero prompt or assertion changes. The same assertion types fail across all evals (digest comments, convention enrichment format, priority/fixVersion inheritance, summary comment). This pattern indicates non-deterministic LLM eval variance rather than regressions from this PR's code changes.

Eval-1 caveat: The prompt change (adding testing readiness template context) may contribute to eval-1's 5 new failures on previously-passing assertions, as the LLM may allocate effort differently with additional instructions. However, these same assertion types also regress in unchanged evals, suggesting the primary cause is variance.


This comment was AI-generated by sdlc-workflow/verify-pr v0.12.0.

@mrizzi mrizzi merged commit fcbfa09 into main Jul 2, 2026
4 checks passed
@mrizzi mrizzi deleted the TC-4878 branch July 2, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant